Class balancing for intent authoring using search

ABSTRACT

Embodiments provide for class balancing for intent authoring using search via: receiving a positive example of an utterance associated with an intent, building an in-intent pool of utterances from a conversation log using the positive example in a first search query of the conversation log; adding the in-intent pool of utterances as a positive class to a training dataset; applying Boolean operators to negate the positive example to form a complement example; building an out-intent pool of utterances from the conversation log using the complement example in a first search query of the conversation log; and adding the out-intent pool of utterances as a complement class to the training dataset. The training dataset may be balanced to include a predefined ratio of positive and complement examples. The training dataset may be used to train or retrain an intent classifier.

BACKGROUND

The present invention relates to training conversational models for usein Natural Language Processing (NLP), and more specifically, tobalancing classifier training data sets. NLP models are trained torecognize the intent of a spoken or written communication that usesunstructured human-language (i.e., an utterance) based on the content,order, and context of words recognized in the communication. A developerindicates various intents that a user is expected to have wheninteracting with the NLP model, and supplies various decision and dialogtrees that may be presented to a user who has indicated a particularintent. As unstructured human language communications may be ambiguous(e.g., due to homologues, colloquialisms, implied meanings, sentenceordering), NLP models include a classifier that is trained to recognizethe intent of an utterance from the available intents specified by thedeveloper. The classifier may determine the intent by various techniquesso that a context-appropriate response is generate to communicationsfrom a user. The dataset used to train the classifier in an NLP model,however, determines the accuracy of a determined intent as much, or morethan the technique chosen to recognize the intent; a classifierdeveloped using a poorly constructed dataset for training will likelyproduce inaccurate results when determining intents in utterances.

SUMMARY

According to one embodiment of the present invention, a method for classbalancing for intent authoring using search is provided, the methodcomprising: receiving a positive example of an utterance associated withan intent, building an in-intent pool of utterances from a conversationlog using the positive example in a first search query of theconversation log; adding the in-intent pool of utterances as a positiveclass to a training dataset; applying Boolean operators to negate thepositive example to form a complement example; building an out-intentpool of utterances from the conversation log using the complementexample in a first search query of the conversation log; and adding theout-intent pool of utterances as a complement class to the trainingdataset.

According to one embodiment of the present invention, acomputer-readable storage medium including instructions that whenexecuted by a processor enable the processor to perform an operation forclass balancing for intent authoring using search is provided, theoperation comprising: receiving a positive example of an utteranceassociated with an intent; building an in-intent pool of utterances froma conversation log using the positive example in a first search query ofthe conversation log; adding the in-intent pool of utterances as apositive class to a training dataset; applying Boolean operators tonegate the positive example to form a complement example; building anout-intent pool of utterances from the conversation log using thecomplement example in a first search query of the conversation log; andadding the out-intent pool of utterances as a complement class to thetraining dataset.

According to one embodiment of the present invention a system for intentauthoring using search is provided, the system, comprising: a processor;and a memory storage device including instructions that when executed bythe processor, enable the processor to: receive a positive example of anutterance associated with an intent; build an in-intent pool ofutterances from a conversation log using the positive example in a firstsearch query of the conversation log; add the in-intent pool ofutterances as a positive class to a training dataset; apply Booleanoperators to negate the positive example to form a complement example;build an out-intent pool of utterances from the conversation log usingthe complement example in a first search query of the conversation log;and add the out-intent pool of utterances as a complement class to thetraining dataset.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 depicts a cloud computing environment according to an embodimentof the present disclosure.

FIG. 2 depicts abstraction model layers according to an embodiment ofthe present disclosure.

FIG. 3 illustrates a computing device for class balancing for intentauthoring via search, according to embodiments of the presentdisclosure.

FIG. 4 illustrates a flow for building a dataset for intent authoring,according to embodiments of the present disclosure

FIG. 5 is a flowchart of a method for class balancing for intentauthoring via search, according to embodiments of the presentdisclosure.

DETAILED DESCRIPTION

When training a classifier for use in a Natural Language Processing(NLP) model, such as a chatbot, to identify various intents that a usermay express, a developer should provide a large and varied trainingdataset to the classifier. A large and varied training set, withexamples that positively indicate a given intent and that do notindicate a given intent allows the classifier to identify features withan utterance that are positively associated with the given intent, areimmaterial to association with the given intent, and that are negativelyassociated with the given intent.

As will be appreciated, a lexical feature that has one association(positive/negative/neutral) with one intent may have a differentassociation with another intent. For example, the word ‘weather’ may bepositively associated with an intent to provide a forecast, andnegatively associated with an intent to provide cooking instructions,whereas the word ‘temperature’ may be positively associated with bothintents. As such, a classifier presented with an utterance that includesboth ‘temperature’ and ‘weather’ may be more likely to identify theintent of the user to receive a forecast than if an utterance using only‘temperature’. In another example, the word ‘recent’ may be positivelyassociated with an intent to identify items having occurred within apredefined time window and negatively associated with an intent toidentify items outside of a predefined time window.

Given the semantic flexibility available in a natural language toformulate a query in different ways, providing a sufficiently largetraining dataset to cover the myriad different formations that differentusers may apply to indicate the same intent is challenging. Particularlychallenging can be providing negative examples, where an utteranceshould be understood to not indicate a particular intent. Withoutproviding sufficient negative examples (also referred to as complementsor complementary examples) to the positive examples, a classifier may bemiss-trained; creating nonsensical associations between utterances andintents. For example, the utterance of “How do I account for daylightsavings time?” may cause a classifier to identify an intent of “AdjustAccount Settings” if the classifier has not been provided this utterance(or similar utterances) as a negative example during training, despite ahuman user being readily able to understand that the query has nothingto do with account settings.

The present disclosure provides improvements for computing devices ingenerating and populating training datasets that provide more accurateand broader complement examples. Using existing chat logs, the trainingdataset is built to include a pool of negative examples and positiveexamples. The negative examples are found from the chat logs using thepositive examples as a basis for a Boolean search. The presentdisclosure balances the two pools and ensures proper classification ofexamples used in the various pools to further improve the reliability ofthe training dataset, and the resulting accuracy of the models trainedtherefrom.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows: On-demand self-service: a cloud consumercan unilaterally provision computing capabilities, such as server timeand network storage, as needed automatically without requiring humaninteraction with the service's provider. Broad network access:capabilities are available over a network and accessed through standardmechanisms that promote use by heterogeneous thin or thick clientplatforms (e.g., mobile phones, laptops, and PDAs). Resource pooling:the provider's computing resources are pooled to serve multipleconsumers using a multi-tenant model, with different physical andvirtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter). Rapid elasticity:capabilities can be rapidly and elastically provisioned, in some casesautomatically, to quickly scale out and rapidly released to quicklyscale in. To the consumer, the capabilities available for provisioningoften appear to be unlimited and can be purchased in any quantity at anytime. Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows: Software as a Service (SaaS): thecapability provided to the consumer is to use the provider'sapplications running on a cloud infrastructure. The applications areaccessible from various client devices through a thin client interfacesuch as a web browser (e.g., web-based e-mail). The consumer does notmanage or control the underlying cloud infrastructure including network,servers, operating systems, storage, or even individual applicationcapabilities, with the possible exception of limited user-specificapplication configuration settings. Platform as a Service (PaaS): thecapability provided to the consumer is to deploy onto the cloudinfrastructure consumer-created or acquired applications created usingprogramming languages and tools supported by the provider. The consumerdoes not manage or control the underlying cloud infrastructure includingnetworks, servers, operating systems, or storage, but has control overthe deployed applications and possibly application hosting environmentconfigurations. Infrastructure as a Service (IaaS): the capabilityprovided to the consumer is to provision processing, storage, networks,and other fundamental computing resources where the consumer is able todeploy and run arbitrary software, which can include operating systemsand applications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows: Private cloud: the cloudinfrastructure is operated solely for an organization. It may be managedby the organization or a third party and may exist on-premises oroff-premises. Community cloud: the cloud infrastructure is shared byseveral organizations and supports a specific community that has sharedconcerns (e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises. Public cloud: the cloudinfrastructure is made available to the general public or a largeindustry group and is owned by an organization selling cloud services.Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 1, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 1 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 2, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 1) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 2 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and class balancing training datasets forintent authoring using search 96.

FIG. 3 illustrates a computing system 300, such as a node 54, which maybe a personal computer, a laptop, a tablet, a smartphone, etc. As shown,the computing system 300 includes, without limitation, a centralprocessing unit (CPU) 350, a network interface 330, an interconnect 340,a memory 360, and storage 370. The computing system 300 may also includean I/O device interface 320 connecting I/O devices 310 (e.g., keyboard,display and mouse devices) to the computing system 300.

The CPU 350 retrieves and executes programming instructions stored inthe memory 360. Similarly, the CPU 350 stores and retrieves applicationdata residing in the memory 360. The interconnect 340 facilitatestransmission, such as of programming instructions and application data,between the CPU 350, I/O device interface 320, storage 370, networkinterface 340, and memory 360. CPU 350 is included to be representativeof a single CPU, multiple CPUs, a single CPU having multiple processingcores, and the like. And the memory 360 is generally included to berepresentative of a random access memory. The storage 370 may be a diskdrive storage device. Although shown as a single unit, the storage 370may be a combination of fixed and/or removable storage devices, such asmagnetic disk drives, flash drives, removable memory cards or opticalstorage, network attached storage (NAS), or a storage area-network(SAN). The storage 370 may include both local storage devices and remotestorage devices accessible via the network interface 330. One or morechat log databases 371 are maintained in the storage 370, which includevarious historical natural language utterances and responses (i.e.,conversations) received from users. The conversations may includehuman-to-human interactions as well as human-to-chatbot interactions.

Further, computing system 300 is included to be representative of aphysical computing system as well as virtual machine instances hosted ona set of underlying physical computing systems. Further still, althoughshown as a single computing system, one of ordinary skill in the artwill recognized that the components of the computing system 300 shown inFIG. 3 may be distributed across multiple computing systems connected bya data communications network.

As shown, the memory 360 includes an operating system 361 (e.g.,Microsoft's WINDOWS® Operating System), at least one NLP Model 362,including a classifier 363, and a trainer 364, including a datasetbuilder 365. The trainer 364 develops one or more NLP models 362, suchas may be used by chatbots, that are trained with an associatedclassifier 363 to identify various intents from user utterances whendeployed. A developer for the NLP model 362 may specify a set of intentsthat utterances from a user are to be mapped to by the classifier 363,and may provide various examples for use in training the classifier 363and the NLP model 362.

The dataset builder 365 may include a search service or an ApplicationProgram Interface (API) to an existing search service to access the chatlog database 371. Using one or more utterances provided by a developer,the dataset builder 365 queries the chat log database 371 for otherexamples based on the provided examples that may be used in a trainingdataset. The utterances stored in the chat log database 371 may beuseful as positive examples for a particular intent, or asnegative/complement examples for a particular intent, and the datasetbuilder 365 identifies which utterances stored in the chat log database371 to include or exclude from various pools in the training dataset.Some utterances found in the chat log database 371 may be useful as bothpositive examples and as negative/complement examples, albeit fordifferent intents.

FIG. 4 illustrates a flow for building a dataset 460 for intentauthoring, which may be understood in conjunction with the method 500illustrated in FIG. 5. A developer submits, via a node 54, one or morepositive examples 410 of an utterance that should be associated with aparticular intent by a classifier 363 (Block 510). The developer mayprovide several positive examples 410, each of which may be labeled toindicate one of several potential intents that the particular positiveexample 410 is associated with.

The dataset builder 365 receives the positive example 410, and queriesthe chat log database 371 via a search engine to identify utterancesfrom historic conversations that are similar to the positive example 410(Block 520). In various embodiments, the dataset builder 365 builds asearch query using the positive example 410 as a basis that removessuperfluous words from the positive example 410, leaving only key termsto be searched. For example, a query of “How do I reset my password” maybe simplified to the key terms of ‘reset’ and ‘password’ when searchingthe chat log database 371. In various embodiments, the developerspecifies the positive example 410 in a search format rather than anatural language format. For example, the developer may submit thepositive example 410 as “reset AND password” to indicate that thedataset builder 365 is to search the chat log database 371 forutterances including both the word ‘reset’ and the word ‘password’.

The dataset builder 365 receives a plurality of results from the chatlog database 371 that are responsive to the query based on the positiveexample 410, and builds an in-intent pool 430 for use in a trainingdataset from those results (Block 530). The in-intent pool 430 includesseveral utterances, specified by the developer or retrieved as resultsfrom the chat log database 371 that match the intent associated with thepositive example 410. In various embodiments, the dataset builder 365presents some or all of the results to the developer to approve of whichresults to include in the in-intent pool 430, while in other embodimentsthe dataset builder 365 automatically selects which results to include.In some embodiments, the developer may specify the in-intent pool 430 toinclude at least a first number of utterances and no more than a secondnumber of utterances, and the dataset builder 365 prunes the result setto provide a number of utterances between the first and second numbersto make up the in-intent pool 430. In various embodiments, the datasetbuilder 365 randomly selects which utterances from the result set toinclude/exclude, while in other embodiments the dataset builder 365 mayrank the results to provide the top N results, the N most uniqueresults, N results gathered evenly over a time window, etc. to make upthe in-intent pool 430.

The dataset builder 365 negates the positive example 410 to form acomplement query 420 (Block 540). In some embodiments, the datasetbuilder 365 negates not only the positive example 410 received from thedeveloper, but also the contents of the in-intent pool 430 to createadditional complement queries 420. When forming the complementquery(ies) 420, the dataset builder 365 may remove superfluous wordsfrom the positive example(s) 410, leaving only key terms to be searched.For example, an utterance of “How do I reset my password” may besimplified to the key terms of ‘reset’ and ‘password’ when searching thechat log database 371. In one embodiment, the dataset builder 365applies a Boolean NOT operator to each key term from a positive example410 to form the complement query 420. For example, the key terms of‘reset’ and ‘password’ may be presented as ‘NOT(reset)’ and‘NOT(password)’. In one embodiment, the dataset builder 365 joins eachkey term via a Boolean OR operator and negates the joined terms via oneBoolean NOT operator. For example, the key terms of ‘reset’ and‘password’ may be joined as ‘reset OR password’ and negated as‘NOT(reset OR password)’. The dataset builder 365 then queries the chatlog database 371 via a search engine to identify utterances fromhistoric conversations that are similar to the complement query(ies) 420(Block 550).

The dataset builder 365 receives a plurality of results from the chatlog database 371 that are responsive to the complement query(ies) 420,and builds an out-intent pool 440 for use in a training dataset fromthose results (Block 560). The out-intent pool 440 includes severalutterances retrieved as results from the chat log database 371 that donot match the intent associated with the positive example 410 (i.e.,that are negative/complement examples). The dataset builder 365 mayselect which complement utterances to include in the out-intent pool 440from the candidate utterances in the results based on a ranking ormerger (e.g., an intersection or a vote) that scores the highest hitsfor the search terms among the results. To ensure that instances ofutterances in the in-intent pool 430 and the out-intent pool do notintersect, the dataset builder 365 may remove any instances of anutterance from the out-intent pool 440 that are also found in thein-intent pool 430 or remove any instances of an utterance from thein-intent pool 430 that are also found in the out-intent pool 440 for aparticular intent.

A balancer module 450 of the dataset builder 365 balances a number ofutterances in the in-intent pool 430 and the out-intent pool 440 tomatch a defined ratio for the training dataset 460 or a defined number Xentries in the in-intent pool 430 and a defined number Y entries in theout-intent pool 440 (Block 570). For example, with an x:y ratiospecified, the balancer module 450 down samples the larger of thein-intent pool 430 and the out-intent pool 440 by removing utterancesfrom the larger pool to leave x utterances in the in-intent pool 430 andfor every y references in the out-intent pool 440. The balancer module450 respects sizing constraints for the size of the in-intent pool 430and the out-intent pool 440 when balancing the pools, and seeks tomaximize the size of the pools within the specified ratio and sizeconstraints for the pools. For example, given the option of providing atraining dataset 460 (according to the ratio of x:y) with X in-intentutterances and Y out-intent utterances or 2X in-intent utterances and 2Yout-intent utterances, the balancer module 450 provides the pools with2X in-intent utterances and 2Y out-intent utterances.

In embodiments where the out-intent pool 440 is the larger pool, thebalancer module 450 may randomly select which utterances toinclude/exclude to meet the size requirement or may determine the Yentries that appear in the lowest number of in-intent pools 430 forother intents to include in the out-intent pool 440. For example, wherea developer specifies five intents, five in-intent pools 430 are created(one for each intent) and five out-intent pools 440 are created (one foreach intent). The balancer module 450 may select which utterances ineach of the out-intent pools 440 based on which utterances appear in thefewest in-intent pools 430 or are least similar to sentences in otherin-intent pools; prioritizing out-intent utterances that do not relateto any of the intents over out-intent utterances that do not relate tosome but not all of the intents.

The dataset builder 365 adds the out-intent pool 440 as a counter classto the dataset 460 and adds the in-intent pool 430 as a positive classto the dataset 460 (Block 580). The dataset builder 365 may continuebuilding the dataset 460 so that each intent includes at least apredefined number of positive and negative/complement examples ofutterances that a classifier 363 may be trained by. The dataset builder365 provides the dataset 460 to the trainer 364 when complete, and basedon a developer request, the trainer 364 trains (or retrains) theclassifier 363 using the dataset 460 with a set of positive andnegative/complement examples extracted from historic conversations(Block 590).

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

In the following, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practicecontemplated embodiments. Furthermore, although embodiments disclosedherein may achieve advantages over other possible solutions or over theprior art, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the scope of the present disclosure. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

Aspects of the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, microcode, etc.) or an embodiment combiningsoftware and hardware aspects that may all generally be referred toherein as a “circuit,” “module” or “system.”

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method comprising: receiving a positive exampleof an utterance associated with an intent; building an in-intent pool ofutterances from a conversation log using the positive example in a firstsearch query of the conversation log; adding the in-intent pool ofutterances as a positive class to a training dataset; applying Booleanoperators to negate the positive example to form a complement example;building an out-intent pool of utterances from the conversation logusing the complement example in a first search query of the conversationlog; and adding the out-intent pool of utterances as a complement classto the training dataset.
 2. The method of claim 1, further comprising:downsampling a larger of the in-intent pool and the out-intent pool tobalance a number utterances between the in-intent pool and theout-intent pool.
 3. The method of claim 1, wherein negating the positiveexample further comprises: extracting key terms from the positiveexample; and applying a Boolean NOT operator to each of the key terms;and wherein building the out-intent pool of utterances furthercomprises: selecting complement utterances for inclusion in theout-intent pool from candidate utterances returned as results to thefirst search query based on a ranking of hits to the first search queryfor in the candidate utterances.
 4. The method of claim 1, whereinnegating the positive example further comprises: extracting key termsfrom the positive example; joining the key terms via Boolean ORoperators; and applying one Boolean NOT operator to the joined keyterms.
 5. The method of claim 1, wherein the out-intent pool of thetraining dataset contains no intersecting utterances with the in-intentpool, further comprising, prior to finalizing the training dataset, foreach utterance in the out-intent pool: in response to determining thatan instance of a given utterance is included in the in-intent pool andthe out-intent pool, removing the instance of the given utterance fromthe out-intent pool.
 6. The method of claim 5, further comprising:wherein utterances comprising the out-intent pool of utterances ascomplement examples for a first intent are included as candidatepositive examples for use in an in-intent pool for a second intentdifferent from the first intent.
 7. The method of claim 1, furthercomprising: training a classifier for a Natural Language Processingmodel via the training dataset.
 8. A computer-readable storage mediumincluding instructions that when executed by a processor enable theprocessor to perform an operation, the operation comprising: receiving apositive example of an utterance associated with an intent; building anin-intent pool of utterances from a conversation log using the positiveexample in a first search query of the conversation log; adding thein-intent pool of utterances as a positive class to a training dataset;applying Boolean operators to negate the positive example to form acomplement example; building an out-intent pool of utterances from theconversation log using the complement example in a first search query ofthe conversation log; and adding the out-intent pool of utterances as acomplement class to the training dataset.
 9. The computer-readablestorage medium of claim 8, further comprising: downsampling a larger ofthe in-intent pool and the out-intent pool to balance a numberutterances between the in-intent pool and the out-intent pool.
 10. Thecomputer-readable storage medium of claim 8, wherein negating thepositive example further comprises: extracting key terms from thepositive example; and applying a Boolean NOT operator to each of the keyterms; and wherein building the out-intent pool of utterances furthercomprises: selecting complement utterances for inclusion in theout-intent pool from candidate utterances returned as results to thefirst search query based on a ranking of hits to the first search queryfor in the candidate utterances.
 11. The computer-readable storagemedium of claim 8, wherein negating the positive example furthercomprises: extracting key terms from the positive example; joining thekey terms via Boolean OR operators; and applying one Boolean NOToperator to the joined key terms.
 12. The computer-readable storagemedium of claim 8, wherein the out-intent pool of the training datasetcontains no intersecting utterances with the in-intent pool, furthercomprising, prior to finalizing the training dataset, for each utterancein the out-intent pool: in response to determining that an instance of agiven utterance is included in the in-intent pool and the out-intentpool, removing the instance of the given utterance from the out-intentpool.
 13. The computer-readable storage medium of claim 12, furthercomprising: wherein utterances comprising the out-intent pool ofutterances as complement examples for a first intent are included ascandidate positive examples for use in an in-intent pool for a secondintent different from the first intent.
 14. The computer-readablestorage medium of claim 8, further comprising: training a classifier fora Natural Language Processing model via the training dataset.
 15. Asystem, comprising: a processor; and a memory storage device includinginstructions that when executed by the processor, enable the processorto: receive a positive example of an utterance associated with anintent; build an in-intent pool of utterances from a conversation logusing the positive example in a first search query of the conversationlog; add the in-intent pool of utterances as a positive class to atraining dataset; apply Boolean operators to negate the positive exampleto form a complement example; build an out-intent pool of utterancesfrom the conversation log using the complement example in a first searchquery of the conversation log; and add the out-intent pool of utterancesas a complement class to the training dataset.
 16. The system of claim15, wherein the instructions, when executed by the processor, furtherenable the processor to: downsample a larger of the in-intent pool andthe out-intent pool to balance a number utterances between the in-intentpool and the out-intent pool.
 17. The system of claim 15, wherein tonegate the positive example, the processor: extracts key terms from thepositive example; and applies a Boolean NOT operator to each of the keyterms; and wherein to build the out-intent pool of utterances, theprocessors: selects complement utterances for inclusion in theout-intent pool from candidate utterances returned as results to thefirst search query based on a ranking of hits to the first search queryfor in the candidate utterances.
 18. The system of claim 15, wherein tonegate the positive example, the processor: extracts key terms from thepositive example; joins the key terms via Boolean OR operators; andapplies one Boolean NOT operator to the joined key terms.
 19. The systemof claim 15, wherein the out-intent pool of the training datasetcontains no intersecting utterances with the in-intent pool, furthercomprising, prior to finalizing the training dataset, for each utterancein the out-intent pool: in response to determining that an instance of agiven utterance is included in the in-intent pool and the out-intentpool, removing the instance of the given utterance from the out-intentpool.
 20. The system of claim 15, wherein the instructions, whenexecuted by the processor, further enable the processor to: train aclassifier for a Natural Language Processing model via the trainingdataset.